Lemma 4: Haptic Input + Auditory Display = Musical Instrument?

In this paper we look at some of the design issues that affect the success of multimodal displays that combine acoustic and haptic modalities. First, issues affecting successful sonification design are explored and suggestions are made about how the language of electroacoustic music can assist. Next, haptic interaction is introduced in the light of this discussion, particularly focusing on the roles of gesture and mimesis. Finally, some observations are made regarding some of the issues that arise when the haptic and acoustic modalities are combined in the interface. This paper looks at examples of where auditory and haptic interaction have been successfully combined beyond the strict confines of the human-computer application interface (musical instruments in particular) and discusses lessons that may be drawn from these domains and applied to the world of multimodal human-computer interaction. The argument is made that combined haptic-auditory interaction schemes can be thought of as musical instruments and some of the possible ramifications of this are raised. 1 Multimodal Challenges Little research has been conducted into how haptic and auditory modalities can best be combined in human-computer interaction. However, in the noncomputing world the two modalities have been partners for a long time. First Hollywood gave us movies (the visual display in computing terms). Then came the talkies (and auditory display in computing). In Brave New World (1932) Aldous Huxley offered us the notion of the feelies – the arm rests in theatre seats would provide haptic stimulation during erotic features. Whilst the feelies are not with us yet haptic displays are making inroads into the development of human-computer interfaces. As the auditory display community has discovered there are several hurdles that must be jumped before a new interaction modality is considered acceptable. Haptic interaction also raises its own usability issues which are being dealt with by researchers. However, the combined use of auditory display and haptic input and output in a single application raises a new set of design challenges. The field is, perhaps, in an analogous position to that of